[SPARK-40733][SQL] Make the contents of `SERDEPROPERTIES` in the result of `ShowCreateTableAsSerdeCommand` have a fixed order by LuciferYang · Pull Request #38238 · apache/spark

LuciferYang · 2022-10-13T11:26:00Z

What changes were proposed in this pull request?

This pr add a sort operation to make the contents of SERDEPROPERTIES in the result of ShowCreateTableAsSerdeCommand have a fixed order.

Why are the changes needed?

Improve Java version compatibility, make the results of ShowCreateTableAsSerdeCommand consistent when using Java version.

Run the following command with Java 19:

mvn clean install -DskipTests -pl sql/hive -am  
mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite

There are 2 failed test:

- SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."
- SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."

The content of SERDEPROPERTIES is correct, but the order is different from result running with Java 8/11/17.

Does this PR introduce any user-facing change?

Yes, the SERDEPROPERTIES part of ShowCreateTableAsSerdeCommand results will be displayed in order by key.

How was this patch tested?

Pass Git Hub Actions
Manual test:

Run the following command with Java 19:

mvn clean install -DskipTests -pl sql/hive -am  
mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite

Before

- SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."
- SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED ***
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187)
  Analysis:
  "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..."

After

Run completed in 23 seconds, 930 milliseconds.
Total number of tests run: 58
Suites: completed 2, aborted 0
Tests: succeeded 58, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

LuciferYang · 2022-10-13T11:26:08Z

Test first

LuciferYang · 2022-10-13T12:12:57Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

      builder ++= s"ROW FORMAT SERDE '$serde'\n"

-      val serdeProps = conf.redactOptions(metadata.storage.properties).map {
+      val serdeProps = conf.redactOptions(metadata.storage.properties).toSeq.sortBy(_._1).map {


spark/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

Lines 1202 to 1210 in 1cb8250

private def showDataSourceTableOptions(metadata: CatalogTable, builder: StringBuilder): Unit = {

// For datasource table, there is a provider there in the metadata.

// If it is a Hive table, we already convert its metadata and fill in a provider.

builder ++= s"USING ${metadata.provider.get}\n"

val dataSourceOptions = conf.redactOptions(metadata.storage.properties).toSeq.sortBy(_._1).map {

case (key, value) =>

s"'${escapeSingleQuotedString(key)}' = '${escapeSingleQuotedString(value)}'"

}

showDataSourceTableOptions already do the similar work in #34753

dongjoon-hyun

+1, LGTM. Thank you, @LuciferYang .
Merged to master for Apache Spark 3.4.0.

LuciferYang · 2022-10-14T00:34:47Z

Thanks @dongjoon-hyun

…lt of `ShowCreateTableAsSerdeCommand` have a fixed order ### What changes were proposed in this pull request? This pr add a sort operation to make the contents of `SERDEPROPERTIES` in the result of `ShowCreateTableAsSerdeCommand` have a fixed order. ### Why are the changes needed? Improve Java version compatibility, make the results of `ShowCreateTableAsSerdeCommand` consistent when using Java version. Run the following command with Java 19: ``` mvn clean install -DskipTests -pl sql/hive -am mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite ``` There are 2 failed test: ``` - SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." - SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." ``` The content of `SERDEPROPERTIES` is correct, but the order is different from result running with Java 8/11/17. ### Does this PR introduce _any_ user-facing change? Yes, the `SERDEPROPERTIES` part of `ShowCreateTableAsSerdeCommand` results will be displayed in order by key. ### How was this patch tested? - Pass Git Hub Actions - Manual test: Run the following command with Java 19: ``` mvn clean install -DskipTests -pl sql/hive -am mvn test -pl sql/hive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite ``` **Before** ``` - SHOW CREATE TABLE using Hive V1 catalog V1 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." - SHOW CREATE TABLE using Hive V1 catalog V2 command: hive table with serde info *** FAILED *** "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." did not equal "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." (ShowCreateTableSuite.scala:187) Analysis: "... SERDEPROPERTIES ( '[serialization.format' = '1', 'field.delim' = ',', 'mapkey].delim' = ',') STORE..." -> "... SERDEPROPERTIES ( '[mapkey.delim' = ',', 'serialization.format' = '1', 'field].delim' = ',') STORE..." ``` **After** ``` Run completed in 23 seconds, 930 milliseconds. Total number of tests run: 58 Suites: completed 2, aborted 0 Tests: succeeded 58, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes apache#38238 from LuciferYang/SPARK-40733. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

fix

41f7253

github-actions bot added the SQL label Oct 13, 2022

LuciferYang changed the title ~~[SPARK-40733][SQL] Make the contents of SERDEPROPERTIES in the ShowCreateTableAsSerdeCommand result have a fixed order~~ [SPARK-40733][SQL] Make the contents of SERDEPROPERTIES in the result of ShowCreateTableAsSerdeCommand have a fixed order Oct 13, 2022

LuciferYang commented Oct 13, 2022

View reviewed changes

dongjoon-hyun approved these changes Oct 13, 2022

View reviewed changes

dongjoon-hyun closed this in e0c2d64 Oct 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-40733][SQL] Make the contents of `SERDEPROPERTIES` in the result of `ShowCreateTableAsSerdeCommand` have a fixed order#38238

[SPARK-40733][SQL] Make the contents of `SERDEPROPERTIES` in the result of `ShowCreateTableAsSerdeCommand` have a fixed order#38238
LuciferYang wants to merge 1 commit intoapache:masterfrom
LuciferYang:SPARK-40733

LuciferYang commented Oct 13, 2022 •

edited

Loading

Uh oh!

LuciferYang commented Oct 13, 2022

Uh oh!

LuciferYang Oct 13, 2022

Uh oh!

dongjoon-hyun left a comment

Uh oh!

LuciferYang commented Oct 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	private def showDataSourceTableOptions(metadata: CatalogTable, builder: StringBuilder): Unit = {
	// For datasource table, there is a provider there in the metadata.
	// If it is a Hive table, we already convert its metadata and fill in a provider.
	builder ++= s"USING ${metadata.provider.get}\n"

	val dataSourceOptions = conf.redactOptions(metadata.storage.properties).toSeq.sortBy(_._1).map {
	case (key, value) =>
	s"'${escapeSingleQuotedString(key)}' = '${escapeSingleQuotedString(value)}'"
	}

Conversation

LuciferYang commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

LuciferYang commented Oct 13, 2022

Uh oh!

LuciferYang Oct 13, 2022

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Oct 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuciferYang commented Oct 13, 2022 •

edited

Loading